Merging this PR will trigger the following deployment actions.

Support deployments

Staging deployments

Production deployments

@bsatoriu the images should be pointing to OPS not DIT for prod and we want to add QGIS back in
If you give me write access to your fork I can make the updates, saves me from having to make my own PR today

@bsatoriu the images should be pointing to OPS not DIT for prod and we want to add QGIS back in If you give me write access to your fork I can make the updates, saves me from having to make my own PR today

I added you @grallewellyn

Thanks, Brian! I made the necessary updates

To prevent drift between staging and prod, we want to keep config in common.yaml as much as possible. So the workflow can be:

1. Test thing in staging via staging specific config
2. When they are ready, move them to common.yaml so that's the behavior of both staging and prod

This way we minimize the config that's prod specific, and can use staging to validate issues.

Can you move most of the config to common.yaml than prod.yaml and i'll merge?

This is complete.

@bsatoriu i see the changes in prod.yaml still

Cross posting from Slack, after @bsatoriu nudged me to point out that the changes to prod are actually required.

The primary difference between staging and prod after your latest changes is:

1. env vars set to differentiate between staging and prod
2. images used themselves (prod has tags, staging has 'develop')

In our experience, (2) is often used to 'test' new images before they get rolled out. This is often better achieved by having people type in the image tag into the 'unlisted image' option in prod, and keep staging and prod have the exact same images instead. This lets people test new images in prod without affecting others, and makes sure staging and prod are as close a match as possible, rather than using staging as almost a 'development' instance. In the future for example, if you're experimenting with s3fuse on staging, making sure the images are the same with prod cuts down on a lot of potential issues when ramping up.

So my suggestion is:

1. The env vars should be put in a place that's common to all profiles - singleuser.extraEnv. MAAP_API_HOST seems same for both staging and prod, so can go in common.yaml. WORKSPACE_BUCKET seems different so it can go in the respective staging or prod yaml files (since singleuser.extraEnv is a dict it'll be merged). DOCKERIMAGE_PATH_DEFAULT and DOCKERIMAGE_PATH_BASE_IMAGE seem to be the name of the images used, which is also available as $(JUPYTER_IMAGE) (that's kubernetes syntax - see https://kubernetes.io/docs/tasks/inject-data-application/define-interdependent-environment-variables/) - so you can set those in common.yaml as well. This also allows people to experiment with images using unlisted choice without needing changes here.
2. Keep all the images as tags, and put them in common.yaml. For testing new images, use unlisted choice.

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/19914343488

The deployment is failing because the image mas.dit.maap-project.org/root/maap-workspaces/2i2c/pangeo:develop doesn't exist. Since it fails in staging, prod deployment will not occur.

This is another reason to keep prod and staging images the same - since staging is supposed to catch issues that affect prod. Here staging has caught an issue, and we no longer know if it'll affect prod or not (and vice versa - staging may succeed but prod may fail).

So in the long run, each production environment should have its own staging where the images are the same.

Since the only difference between staging and production is the image tags, the only way the pipeline could fail is if the image doesn't exist. If the image doesn't exist, we can quickly push it and rerun the deployment
Once we get the hang of the 2i2c deployment process, this won't be an issue again

If there is something wrong with the images, then it won't launch in 2i2c but that is a different issue